Deployable Classifiers for Malware Detection

The application of machine learning methods to malware detection has opened up possibilities of generating large number of classifiers that use different kinds of features and learning algorithms. A straightforward way to select the best classifier is to pick the one with best holdout or cross-validation performance. Cross-validation or holdout gives a point estimate of generalization performance that varies with training data and learning algorithm parameters. We propose a classifier selection criterion that considers bounds on the performance estimates using confidence intervals in conjunction with a performance target. Performance targets are commonly used in practice, particularly in security applications like malware detection, for classifier selection. The proposed criterion, called deployability, selects a classifier as deployable if the cost target lies within or above the classifier’s expected cost confidence interval. We conducted an experiment with machine learning based malware detectors to evaluate the criterion. We found that for a given confidence level and cost target, even the classifier with least expected cost may not be deployable and classifiers with higher expected cost may also be deployable.